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ABSTRACT 

Using specialized knowledge and perspectives of a set in decision-makings about issues that are 
qualitative is very helpful. Delphi technique is a group knowledge acquisition method, which is also used for 
qualitative issue decision-makings. Delphi technique can be used for qualitative research that is exploratory 
and identifying the nature and fundamental elements of a phenomenon is a basis for study. It is a structured 
process for collecting data during the successive rounds and group consensus. Despite over a half century of 
using Delphi in scientific and academic studies, there are still several ambiguities about it. The main problem in 
using the Delphi technique is lack of a clear theoretical framework for using this technique. Therefore, this 
study aimed to present a comprehensive theoretical framework for the application of Delphi technique in 
qualitative research. In this theoretical framework, the application and consensus principles of Delphi 
technique in qualitative research were clearly explained. 
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I. INTRODUCTION 

Delphi is a name of temple in ancient Greek, which is the origin of Apollo's divine inspirations, god of 
sun and music. Greeks and ancient nations came in Pythia, the goddess of Delphi temple to predict the future [1] 
[2]. The Delphi technique was originally proposed based on people's conjecture, judgment, and inspiration but 
gradually took the academic form. For the first time in the late 1950s, in a research by U.S. RAND Corporation, 
the Delphi was introduced for the scientific study of experts' opinions on military defense project. However, for 
the security reasons, this technique was not proposed over ten years and in 1963, Dalkey and Helmer introduced 
it [3] [4]. Its first non-military use was suggested for economic development planning [5] [6] [7] .This technique 
gradually found its place in the academic studies but from the mid-90s, it became highly popular. Landeta's 
study indicated that since 1995 to 1999 a total 444 articles on this technique have been published in "Science 
Direct" and "ABI / Inform" journals. Since 2000 to 2004, this number has increased to 667 articles [6]. Among 
the various features of the Delphi technique, its four features are usually unchanged including anonymity, 
iteration, controlled feedback, and statistical "group response" [8] [9] [10]. 

Most researchers have used all [11] or some [12] [13] of the definition set out by Linstone and Turoff 
(1975) who define the Delphi technique as " a method for structuring a group communication process so that the 
process is effective in allowing a group of individuals, as a whole, to deal with a complex problem" (p. 3) [14]. 
The main purpose of the Delphi method is 'to acquire the most reliable consensus of a group of experts' opinion 
by 'a series of intensive questionnaires combined with controlled opinion feedback' (p. 458) [15]. By obtaining 
the consensus of a group of experts using the process, researchers can identify and prioritize issues and develop 
a framework to recognize them [16] [2]. Hasson et al. (2000) argued that modified Delphi, policy Delphi, and 
real-time Delphi have all been used, while there are many potential types of Delphi techniques [17]. Three broad 
categories are generally in use and these are classical, policy and decision [18]. The Delphi technique is applied 
as a tools and method for consensus building using a series of questionnaires for data collection from a panel of 
selected participants [15] [19] [14] [20] [21] [22]. 
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Undoubtedly, the collective wisdom in decision -making can lead to more complete and comprehensive 
decision-making. However, the group problem solving through traditional consensus method is associated with 
numerous difficulties. The group members who have high self-confidence are dominating the weaker members. 
Some are also pressured to conform to group comments and or because they respect other people, they do not 
express their opinions. Hence, a group problem solving approach will be often futile and ineffective. The 
anonymity principle is used to solve this problem. In Delphi technique, the experts and people who are used the 
survey do not know each other. Anonymity ensures overcoming the obstacles of groupthink [23] [24] [25] [1] 
[26]. A coordinator collects the experts' opinions and then he/she provides other members with the summarized 
results. Then, based on the summarized results in the previous step, individuals again adjust and express their 
opinions. Finally, after reaching a consensus, the results are discussed in terms of a statistical report (usually 
mean or median) and are used for decision-making [9] [27] [28]. 

The Delphi technique is a research approach to gain consensus using a series of questionnaires and the provision 
of feedback to participants who have expertise in key areas. The main problem in using the Delphi technique is 
the lack of a clear theoretical framework for using this technique. Therefore, this study aimed to present a 
comprehensive theoretical framework for the application of Delphi technique in qualitative research. In this 
theoretical framework, we clearly explain the application and consensus principles of Delphi technique in 
qualitative research. 

II. DELPHI TECHNIQUE THEORETICAL FRAMEWORK 

The major weakness of Delphi is the lack of a theoretical framework. Delphi as a research 
methodology has been variously presented as a survey, study, procedure, method, approach, polling, and 
technique [14] [29] [30] [31] [24] [25] [32] [33] [11] [34] [35] [36] [37] [38]. Nevertheless, it is generally 
acknowledged that the Delphi methodology is not uniform in its application [17] [1]. In addition, there are 
always ambiguities about the application requirements, panel size, how to select the panel, and recognizing the 
end of Delphi stages. In this comparative study, a framework is presented for the application of the Delphi 
technique in the qualitative decision-makings. 



Delphi technique application requirements 



Composition and panel size of experts 



T 



Gathering expert's opinions 

1 



Determine the level of consensus 



Controlled feedback 




Fig 1. Theoretical framework of Delphi technique in qualitative research 
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2.1 Delphi Technique's Application Requirements 

The most important requirements for the use of Delphi are the need for experts' judgment, group 
consensus to achieve the results, anonymity in data collection, a complex, multidimensional, and 
interdisciplinary problem, lack of consensus and imperfect knowledge, experienced and capable experts, 
dispersion of experts, no time limitation, and lack of cost-effective method [39] [6] [40] [28] [7] . 

In qualitative research that is mainly based on the individuals' judgments and opinions, there is a 
prerequisite for using the Delphi method. If these studies are based on the expert opinions, then the use of 
inferential statistical techniques such as mean tests will not be justified. Since either experts are not so much that 
can be accessed wide samples of them or the time and cost to access them are difficult. In addition, to review the 
experts' opinions this proverb is truer: "you may know by a handful the whole sack". 

Therefore, the two types of qualitative research should be distinguished to use the Delphi technique. 
Delphi technique can be used for qualitative research that is exploratory and identifying the nature and 
fundamental elements of a phenomenon is a basis for the study. Consider the problem of measuring customer 
satisfaction. If you were to examine the customer satisfaction, statistical sampling and inferential statistical 
methods would be used. However, if you decide to measure the customer satisfaction, then experts' opinions and 
the Delphi technique can be used. One of main application of Delphi technique is screening the items in 
operations research problems and of multi -criteria decision-making (MCDM) techniques. 

2.2 Composition and Panel Size 

Essentially, an accurate mechanism does not exist for identifying the number of individuals or the 
number of panels for inclusion in any individual study [41]. Although there are some disagreements about the 
composition and panel size of Delphi technique, a dominant pattern can be detected. It has been recommended 
that the panel size may vary according to the topics covered, the nature of different viewpoints included, and the 
time and money available [18] and it is also suggested to use a combination of individuals with multiple 
specialties and heterogeneous groups better than the homogeneous groups [1] [26]. Hsu and Sandford (2007) 
claimed that, "Delphi subjects should be highly trained and competent within the specialized area of knowledge 
related to the target issue" (p.4) [42]. Hogarth (1978) argued that between six and twelve members are ideal for 
Delphi technique and according to Clayton (1997), if a mixture of experts with different specialties is used, 
between five and ten members are sufficient (p. 5) [26]. While some Delphi studies considered fewer than 10 
members in their panels [43] [44], other studies included more than 100 participants [45] [46]. 

Delphi panel is done with the participation of individuals who have the knowledge and expertise of the 
study subject. These people are known as the Delphi panel. One of the most important phases of Delphi 
technique is selecting eligible members for the Delphi panel because the validity of the results depends on the 
competence and knowledge of panel members [1]. In previous studies, Delphi has not recommended a specific 
method for sampling. Snowball technique can be used for sampling. The snowball sampling is one of the most 
common approaches of sequential sampling. This type of sampling is a non-probability method, which is also 
randomly selected. This method is suitable when the members of a group or community cannot be easily 
identified. In this method, the researcher first identifies some people and after receiving the information, he/she 
wants them to introduce other people [47]. This method can be also used to identify the experts in a particular 
field [48]. 

2.3 Gathering Expert Opinions 

In qualitative research, which aims to determine the importance or screening of items, Likert scale can 
be used to gather the experts' opinions. Five -point or seven-point Likert scale is common. To develop five-point 
Likert scale, two "extremely agree" and "extremely disagree" linguistic scales are used at both ends of a 
spectrum. By defining the intermediate values, this scale can be developed and nine -point scales can be also 
used. 

Diefenbach, Weinstein, and O'Reilly (1993) conducted an investigation of a range of Likert items, 
including 2-point, 5-point, 7-point, 9-point, 11-point, 12-point, and percentage (100-point) varieties [49]. Lewis 
(1993) found that 7-point scales resulted in stronger correlations with t-test results [50]. It was demonstrated that 
"no scale performed significantly better than the seven-point verbal category scale on any criterion" in the two 
studies carried out (p. 189) [49]. 
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Extremely Disagree Nermer agree Agree Extremely 

disagree nor disagree agree 



Fig 2. A five-point Likert scale [51] 



Extremely Mostly disagree Somewhat Nermer agree Somewhat Mostly agree Extremely 

disagree disagree nor disagree agree agree 



Fig 3. A seven-point Likert scale [51] 

After gathering the experts' opinions, the mean score of their opinions on each dimension is calculated. 
Given the theoretical framework, if there is no consensus, the experts will be provided with the calculated mean 
as a controlled feedback with questionnaire. After several rounds, when the consensus was achieved, based on 
the average of the final round, the items are screened. If nine -point scale is used, typically, the criteria with 
mean less than seven are removed. For seven-point and five-point scales, criteria with means lower than five and 
four are respectively removed. 

2.4 Determining the Level of Consensus 

Another problem with the Delphi technique is a scientific method for determining the level of consensus. A variety 
of methods has been proposed in different studies. Some researchers have determined Delphi rounds as a basis for reaching 
agreement. Between 2 and 10 rounds have been reported in various articles [53]. Fan and Cheng have believed that research 
has shown that three rounds are adequate for the Delphi technique [40]. Some also argued that the traditional Delphi contains 
four rounds that researchers have shortened it into two or three rounds to achieve the study's objectives [54] [5]. Such an 
approach is a target displacement. In fact, reaching a consensus is a basis for the end of Delphi rounds while in this approach 
doing a certain number of rounds is regarded as a basis for reaching an agreement. So, another solution must be identified to 
reach consensus. In a study, according to 114 articles about the Delphi technique, von der Gracht (2012) has presented 15 
ways to achieve consensus [9] . 

Kendall's coefficient of concordance (Kendall's W), a consensus criterion representing the level of consensus 
between the participants [55] [56] [57], was calculated along with the mean rank and standard deviation. Kendall's 
coefficient of concordance ranges from 0 to 1, indicating the degree of consensus reached by the panel (strong consensus for 
W> 0.7; moderate consensus for W= 0.5; and weak consensus for W< 0.3) [58]. Kendall's coefficient of concordance is also 
a scale for determining the level of coordination and agreement between several ranks of n phenomenon. This scale shows a 
rank correlation between m ranking set. It can be used to determine the inter-judge reliability. Kendall's coefficient of 
concordance indicates that those who have ordered several categories according to their importance have used the same 
criteria to judge the importance of each category and in this regard, they agree to each other [57]. 
Kendall's coefficient is calculated as follows: 

W = z^l * (1) 

Where 



L = L 



J?l = total ranks of a factor 

m= number of rank sets or judges 

n= number of ranked factors or phenomena 
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The value of this scale during the full coordination or agreement is one and in the absence of perfect 
coordination, it is zero. Schmidt (1997) proposed two statistical criteria to make decisions about stopping or 
continuing the Delphi rounds. The first criteria is a strong consensus among the panel members which is 
determined based on Kendall's coefficient of concordance. If there is no such a consensus, the constant 
coefficient or its negligible growth in two successive rounds shows that consensus has not been increased and 
the survey process must be stopped. Statistically significant coefficient w is not enough to stop a Delphi process. 
For panels with fewer than ten members, very small values of w are also significant [56]. 

III. CONCLUSION 

To summarize, the Delphi technique is a research approach to gain consensus using a series of 
questionnaires and the provision of feedback to participants who have expertise in key areas. This method is 
especially useful when researchers need to collect ideas from isolated experts on a specific topic and establish 
agreement to discover the underlying assumptions or perspectives among the experts. However, the major 
weakness of Delphi is the lack of a theoretical framework. Delphi as a research methodology has been variously 
presented as a survey, study, procedure, method, approach, polling, and technique. Nevertheless, it is generally 
acknowledged that the Delphi methodology is not uniform in its application. 

Although previous studies have always been disadvantaged by the lack of a theoretical framework for 
Delphi technique, considering numerous studies, access to a theoretical framework is simple. Given the findings 
of this study, it can be said that in qualitative research, it is recommended to use a group of ten experts with 
different specialties. The snowball sampling technique can also be used to identify and select the sample. If the 
purpose is to summarize and screen an item set, a nine -point Likert scale can be used to gather the experts' 
opinions. Kendall's coefficient of concordance can be utilized to determine the agreement and end of the Delphi 
technique. Therefore, it can be concluded that the Delphi technique is an appropriate and acceptable mechanism 
to achieve consensus when developing sets of indicators. 
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